Connecting the Dots on NetApp’s AI Data Engine

Poor data management risks that very thing AI depends on: clean, connected, and governed data.  And, it remains the “gift that keeps giving” with wasted capacity, higher spend, potential governance gaps, and now stalled or failed AI projects in a hypercompetitive do-more-with-less world. So what's an org on an AI journey supposed to do?

This year’s NetApp INSIGHT announcements and Tech Field Day presentations centered on how NetApp is bringing AI pipelines closer to the storage layer to help solve some of the challenges that have plagued AI pipelines and also adding tangible benefits around simplifying workflows and minimizing both copies and data movement.

NetApp wants to help solve these problems and herald what they’ve dubbed an “Era of Data-Enabled Intelligence.” Additionally, this unified platform approach builds on ONTAP’s existing strengths and brings in new additions like the AFX cluster and, the focus of this post, its software companion NetApp AI Data Engine. Together, these pieces start to show the shape of NetApp’s AI vision for enterprises.

Here’s how NetApp is structuring this platform. Let’s start with the core components.

• ONTAP: The belle of any NetApp Storage Admin’s heart underpins everything. This storage operating system has been around for 30+ years. Baked into ONTAP is multi-protocol support (file, block, and object). Snapshots, SnapMirror, and FlexClone enable copyless workflows, zero-cost cloning, training set creation, support for movement across environments, and minimize data gravity. All powerful tools even outside of an AI pipeline.
• AFX Cluster Still runs ONTAP, but with true disaggregation. Storage controllers and storage shelves can scale independently.  Additionally, these AFX cluster deployments can include Data Compute Nodes. These are NetApp Appliances, but future plans include third-party hardware (interoperability matrix) to open up options to leverage the latest GPU advancements. These Data Compute Nodes are Linux servers with GPUs that run microservices.  Decoupling storage and compute can offer flexibility, a future-proof architecture, and greater simplicity.
• Metadata Fabric. Doesn’t this feel declarative in the same way Data Fabric did in 2016? NetApp was early to the hybrid cloud game at a time when cloud/on-premises were either/or not today’s yes/and. This “consume ONTAP almost anywhere” capability is a key piece of NetApp’s value proposition. There’s a truism in IT that if you’re doing your job right, no one thinks about you. That’s Data Fabric in a nutshell. It just quietly handled placement until it wasn’t a “thing.” Metadata fabric could be the evolutionary next step: moving from moving data to making sense of it. Honestly, this is the story I’m looking forward to watching unfold
• AI Data Engine (AIDE). Finally, we’ve made it to the star of this post and enabler of the metadata fabric. This console runs as a web application from a Data Compute Node.  But AIDE is way more than just a pretty web UI designed for data scientists. It also runs the following services that make up the core building blocks of an AI pipeline.

These components aren’t just a product stack. It’s a new design pattern for running AI pipelines close to the data layer. Truthfully, I struggled to wrap my head it until I picked up a pencil.

Here’s my rough sketch to try and attemp to connect the dots..

Once I mapped it out, things made more sense in what NetApp is trying to do and in the context of today's AI projects. How’d I do?

Sure, organizations have been running AI and ML workloads for decades. However, AI and LLMs as we know them are still in a liminal stage. The combination of data management, tooling, and pipeline orchestration brings real complexity, and any solution that can simplify that while also unlocking value from the metadata layer is a real advantage. After all, truth and lineage depend on metadata. An AI project without those things is basically worthless.rough sketch of how ONTAP, AFX, Metadata Fabric, and AIDE fit together across the AI pipeline.rough sketch of how ONTAP, AFX, Metadata Fabric, and AIDE fit together across the AI pipeline.

Being able to run this unified AI platform on-premises is a huge advantage at a time when EMEA is leaning real hard into data sovereignty and cloud repatriation. AI pipelines have mostly been the domain of hyperscalers and industry's biggest players, but NetApp's approach feels more accessible to mid-sized enterprises in a way most AI stacks don't.

Originally published on LinkedIn. I’m gradually moving my storage, infrastructure, and career-in-tech writing here so it has a permanent home.