Data Storage Options

Match the format to the job

Before naming a format, name the requirement. A file format should match three things: the type of data being stored, the applications that must read or write it, and whether people need to read the file or systems need efficient storage and processing.

That last pair is the fork in the road. Human-readable formats are not usually optimized for storage space or processing, and formats optimized for storage and processing are not usually pleasant to open in a text editor. You are choosing which of the two you need, not looking for a format that is best at everything.

Delimited text keeps data readable

Comma-separated values, or CSV, stores plain-text fields separated by commas and rows terminated by a new line. The first line can optionally contain the field names.

CustomerID,Name,City
1001,A. Ferrer,Girona
1002,B. Okafor,Lagos

CSV has close relatives: TSV and space-delimited text change the separator, while fixed-width text uses fixed character positions.

Delimited text is suitable for structured data that must be human-readable and accessible to many applications. That is its whole reason to exist: anyone can open it, and almost anything can parse it.

JSON and XML carry hierarchy

JavaScript Object Notation, or JSON, represents objects with attributes and supports hierarchical structure. An attribute can itself contain an object or a collection of objects, which is why JSON handles both structured and semi-structured data.

{ "orderId": 5501,
  "customer": { "id": 1001, "city": "Girona" },
  "lines": [ { "sku": "A-1", "qty": 2 }, { "sku": "B-7", "qty": 1 } ] }

Extensible Markup Language, or XML, is human-readable and uses tags to define elements and attributes. Tags are written inside angle brackets. XML is more verbose than JSON but remains in use in some systems, so recognizing it matters even when a new design would not choose it. Here is the same order in XML:

<order id="5501">
  <customer id="1001"><city>Girona</city></customer>
  <lines>
    <line><sku>A-1</sku><qty>2</qty></line>
    <line><sku>B-7</sku><qty>1</qty></line>
  </lines>
</order>

Some data is not text at all. Files such as images, video, audio, and application-specific documents are stored as raw binary that an application must interpret and render. Data professionals commonly call these files binary large objects, or BLOBs.

Keeping the four text-and-binary families apart is most of the recall this topic asks for.

Optimized formats trade readability for efficiency

When storage space, compression, and processing speed matter more than reading the file by eye, two common optimized formats are Parquet and Avro, and they use different layouts.

Parquet is a columnar format that stores data for each column together within row groups and uses metadata to locate relevant chunks. Because a query can jump to the chunks holding only the columns it needs, Parquet supports efficient compression and processing, including of nested data.

Avro is row-based, describes its schema as JSON in the file header, and holds the data as binary records. Keeping whole records together suits moving data around: Avro is useful for compression and for reducing storage and network bandwidth.

Delta Lake adds transactions to Parquet

Parquet is efficient but, on its own, a set of files. Delta Lake builds on Parquet by adding a transaction log for ACID transactions, data versioning, and reliable updates.

The log is what changes the behavior. Because every change is recorded before it counts, a reader never sees half of a write, an earlier version of the data can still be identified, and an update that fails does not leave the files in an inconsistent state.

Note what it does not do: it adds no fixed-width text layout, no XML tag hierarchy, and no graph model. It adds transactional guarantees to files that already exist.

File stores and databases are the two categories

Above the level of formats sit two broad categories of data store: file stores and databases. Getting the category right comes before choosing anything else.

Important organizational files are commonly kept in a central shared file storage system rather than only on local disks or removable media. Hosting that central location in the cloud can provide cost-effective, secure, and reliable storage for large volumes of data.

A database is a dedicated system for storing, managing, and querying data records rather than files. The distinction is not about size or importance: it is about whether the store keeps files that applications open directly, or records that the system itself manages and answers questions about.

Relational databases organize records in tables

Relational databases commonly store structured data in tables. Each entity instance has a primary key, and other tables can use that key to reference the entity.

That reference is what supports normalization, including avoiding the repeated storage of the same entity details. An order table holds a customer key rather than a duplicate copy of the customer's name, address, and phone number, so a corrected address is corrected once.

The CustomerID in each Order row is a foreign key, because it references the primary key of the Customer table.

Relational tables are managed and queried with Structured Query Language.

Nonrelational models fit other shapes

Nonrelational databases do not apply a relational schema to their data. They are a family, not one alternative, and each member matches a different record shape.

Read the table as a set of questions about the data. Is a record just a value fetched by its key? Is it a document with its own fields? Are its columns naturally grouped? Or is what matters most the connections between entities?

That last question is the graph model's whole purpose. When data is highly connected, and the connections themselves are the thing being queried, a graph database represents them directly rather than reconstructing them from keys at query time.

The contrast with its neighbors is what makes the choice recognizable:

• A document database parses each value as a JSON document, but it does not record links between documents as first-class data.
• A column-family database groups related columns; the grouping says nothing about relationships between entities.
• A relational database can connect records through key values in tables instead of graph nodes and links.

Match common use cases to Azure datastores

Start with what the workload must store and how applications must access it, then select the Azure service whose documented behavior matches those constraints.

Choose a file store when applications need to work with files directly: human-readable exports, optimized analytical files, or binary media. Choose a database when the workload needs a dedicated system to manage and query records. Within databases, choose a relational design for structured tables, which can use keys to connect records. Choose a nonrelational model for key-value, JSON document, column-family, or graph-shaped records.

Use the category to narrow the choice, but finish with a named Azure datastore when the question asks for one.

The common mappings are concrete:

The protocol and data model decide the service. A legacy application that already opens shared directories over SMB points to Azure Files; a collection of unstructured objects points to Blob Storage instead.

Key takeaways

• Choose a format from the requirement: the data type, the applications that read or write it, and whether people must read the file or systems need storage and processing efficiency.
• Know the four families: CSV and its delimited relatives for readable structured rows, JSON for hierarchical objects, XML for tag-defined elements, and raw binary large objects for media and application documents.
• Optimized formats split by layout: Parquet is columnar with metadata over row groups, Avro is row-based with a JSON schema in its header and binary records, and Delta Lake adds a transaction log for ACID transactions, versioning, and reliable updates.
• Two store categories: file stores keep files; databases manage and query records.
• Within databases: relational tables store records and can use keys to connect them. SQL queries these tables; key-value, document, column-family, and graph models fit other record shapes.
• Name the Azure datastore: Azure Files serves managed SMB/NFS shares, Blob Storage holds unstructured objects and data lakes, Azure Tables stores schemaless key/attribute data, Azure SQL serves relational records, and Cosmos DB serves globally distributed nonrelational workloads.