Damir's Corner.

Elasticsearch log ingest config in .NET

When trying to get the logging to Elasticsearch from .NET to work in the production environment, I spent too much time configuring the logger. It was mostly due to missing and incorrect documentation.

By default, all log entries are sent to the logs-dotnet-default data stream. And there isn't much information about ingest configuration in the official documentation. There is no documented way to configure a data stream. Supposedly, you can only specify a format string for the index to use:

{
  "Logging": {
    "Elasticsearch": {
      "Index": "dotnet-{0:yyyy.MM.dd}"
    }
  }
}

However, setting its value in my configuration file didn't seem to have any effect whatsoever. The log entries were still being sent to the logs-dotnet-default data stream. As I was getting desperate, I tried to configure the logger in code. With more success:

• I could configure the index directly:

  builder.Logging.AddElasticsearch(options =>
      options.Index = new IndexNameOptions { Format = "logs-myapp-development" }
  );
  

• But I could also configure a data stream:

  builder.Logging.AddElasticsearch(options =>
      options.DataStream = new DataStreamNameOptions
      {
          Type = "logs",
          DataSet = "myapp",
          Namespace = "development"
      }
  );
  

This made me wonder why it wouldn't work from the configuration file. I went digging into source code. And it turned out that both index and data stream can be specified in the configuration file:

• Working index configuration must be different from what is documented (notice the extra Format property inside index):

  {
    "Logging": {
      "Elasticsearch": {
        "Index": {
          "Format": "logs-myapp-development"
        }
      }
    }
  }
  

• And the data stream configuration is also supported, just missing from the documentation:

  {
    "Logging": {
      "Elasticsearch": {
        "DataStream": {
          "Type": "logs",
          "DataSet": "myapp",
          "Namespace": "development"
        }
    }
  }
  

When both Index and DataStream are set, Index has precedence, and DataStream configuration is ignored.

There's also an important difference between the two configurations. Data stream configuration only works for data streams, which follow the naming scheme convention: {type}-{dataset}-{namespace}. The three data stream configuration fields Type, DataSet, and Namespace get concatenated together into the final data stream name. If you omit any of them from the configuration, they will fall back to their default values:

• type: logs
• data set: dotnet
• namespace: default

Index configuration allows you to use data streams with names, which don't match the naming scheme. If you have previously created an Elasticsearch data stream named myapp-development (i.e., without the type part), you can send logs to it with the following configuration.

{
  "Logging": {
    "Elasticsearch": {
      "Index": {
        "Format": "myapp-development"
      }
    }
  }
}

If you authenticate with sufficient permissions, this will work even when the data stream is not yet created, as long as there is a matching index template. Without one, an index with the specified name will be created instead. It's similar to how (with sufficient permissions, of course) new data streams of type logs will be created on the fly since there's a matching index template preconfigured by default.

This is another one of my blog posts written for future me. As always, I hope it will help other as well, so they will spend less time figuring out the correct configuration for sending logs to Elasticsearch according to administrator's requirements.