Generate C# models from Protobuf proto files directly from Visual Studio

Generating C# classes inside .NET Core project from proto using protogen

I wrote about boosting performances by choosing the right serialization in .NET projects. The top best performing according to my tests and many of the online comparisons are Apache Avro and Google Protocol Buffer (Protobuf) serialization. If you have anything to do with any of the Google's service you might have to deal with Google Protobuf probably.

There is no hassle if you are sharing your data between two .NET applications since on both sides you will probably share library with models which are serialized/deserialized in your solution. The problem occures if you need to do cross platform model sharing. Well, problem might be to harsh word to use, let's say it is not working out of the box by just defining models :)

Protobuf, like Apache Avro supports data description format which is definition of the data structure. From this you can generate models which will bind to deserialized data and serialize from the language specific objects. For Protobuf these are *.proto files. Here is a sample of *.proto files which comes with protobuf-net project solution

package people;
message person { 
  required string name = 1; 
  optional int32 id = 2; 
  optional string email = 3; 
  enum phone_type { 
    mobile = 0; 
    home = 1; 
    work = 2; 
  } 
  message phone_number { 
    required string number = 1; 
    optional phone_type type = 2 [default = home]; 
  } 
  repeated phone_number phone = 4; 
  repeated int32 test_packed = 5 [packed = true];
  optional int32 test_deprecated = 6 [deprecated = true];
  optional int32 foreach = 7;
}
message opaque_message_list { repeated bytes messages_list=1; }

So the structure is not to complicated and you can easily build your own proto to share it across the platforms and languages to be used, but it is not so easy in case you have to build your own model based on a complex proto file. Google does provide documentation for generating C# classes from proto files https://developers.google.com/protocol-buffers/docs/reference/csharp-generated but for some reasons I kept getting errors and simple task suddenly turned into a nightmare. Surprisingly, there are many opened questions on this topic but not so many clear answers how to generate C# models from proto files.

Eventually I made it work and wrapped it up in a demo .NET Core project and automated to generate C# models on pre-build action in Visual Studio. So let's cut the talk and switch to Visual Studio and code.

Setting up the project

I decided to do a demo in .NET Core project instead of .NET 4.x initially because of the project structure and approach difference for .csproj file. In .NET 4.x files that are part of the project are explicitly listed in .csproj file. In .Net Core project, all files inside project folder are considered as a part of the project, you list excluded files in your .csproj.

This means once the C# model is generated from proto file, it will become part of the project, while in .NET 4.x you will have to explicitly add it in your .csproj file, which is just an additional hassle. Ok, so now when you know why I decided to do it for .NET Core, you can create a new .NET Core project (any project template, although for test I picked simple .NET Core console app) and add protobuf-new nuget package from package management console or .NET Core CLI, which ever you prefer.

I stick to PM console

Install-Package protobuf-net -Version 2.3.13

Now when we have protobuf-net nuget package installed, we need to add protogen tool which is part of protibuf-net project, but unfortunately does not come with the nuget package installation, which means we need to got to GitHub protobuf-project release page, download the latest release and build it with Visual Studio so we can get the protogen tool binaries available in our solution so we can setup the pr-build action and generate code from the proto.

Tool protogen will be compiled for both .NET Core cross-platform as .dll and .NET 4.x as .exe file. Since I decided to got with .NET Core, I used the dll one for .NET Core cross-platform and copied it in my solution folder under $(SolutionDir)tools\protogen

Now we are ready to add before-build action to prepare our classes from the proto files.

Setting up BeforeBuild action in csproj file

Now that we have protogen tool part of our solution, we can invoke it from Visual Studio by adding step in csproj file which will be triggered on BeforeBuild action. This is how it should look like inside the csproj file XML

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="protobuf-net" Version="2.3.13" />
  </ItemGroup>

  <Target Name="protogen" BeforeTargets="BeforeBuild">
    <Exec Command="dotnet $(SolutionDir)tools\protogen\protogen.dll Person.proto --proto_path=$(ProjectDir)Protos --csharp_out=$(ProjectDir)Models --package=$(ProjectName)Models" />
  </Target>

</Project>

The Target/Exec node contains the Command attribute which is the command that will be execute before the build happends in Visual Studio. To test it we need a sample proto file. I used the same file which comes with protobuf-net solution I downloaded from GitHub in order to build protogen but with one small change, I removed the package definition line. This is because I want to set my namespace from the BeforeBuild action.

message Person { 
  required string name = 1; 
  optional int32 id = 2; 
  optional string email = 3; 
  enum phone_type { 
    mobile = 0; 
    home = 1; 
    work = 2; 
  } 
  message phone_number { 
    required string number = 1; 
    optional phone_type type = 2 [default = home]; 
  } 
  repeated phone_number phone = 4; 
  repeated int32 test_packed = 5 [packed = true];
  optional int32 test_deprecated = 6 [deprecated = true];
  optional int32 foreach = 7;
}
message opaque_message_list { repeated bytes messages_list=1; }

There is also Models folder added to the project structure and this one will contain the output of the BeforeBuild action. Now just build the project and you should have Person.cs in your Models folder which should look something like this

// This file was generated by a tool; you should avoid making direct changes.
// Consider using 'partial classes' to extend these types
// Input: Person.proto

#pragma warning disable CS1591, CS0612, CS3021, IDE1006
namespace Protobuf.Autogenerate.Sample.Models.Models
{

    [global::ProtoBuf.ProtoContract()]
    public partial class Person : global::ProtoBuf.IExtensible
    {
        private global::ProtoBuf.IExtension __pbn__extensionData;
        global::ProtoBuf.IExtension global::ProtoBuf.IExtensible.GetExtensionObject(bool createIfMissing)
            => global::ProtoBuf.Extensible.GetExtensionObject(ref __pbn__extensionData, createIfMissing);

        [global::ProtoBuf.ProtoMember(1, Name = @"name", IsRequired = true)]
        public string Name { get; set; }

        [global::ProtoBuf.ProtoMember(2, Name = @"id")]
        public int Id
        {
            get { return __pbn__Id.GetValueOrDefault(); }
            set { __pbn__Id = value; }
        }
        public bool ShouldSerializeId() => __pbn__Id != null;
        public void ResetId() => __pbn__Id = null;
        private int? __pbn__Id;

        [global::ProtoBuf.ProtoMember(3, Name = @"email")]
        [global::System.ComponentModel.DefaultValue("")]
        public string Email
        {
            get { return __pbn__Email ?? ""; }
            set { __pbn__Email = value; }
        }
        public bool ShouldSerializeEmail() => __pbn__Email != null;
        public void ResetEmail() => __pbn__Email = null;
        private string __pbn__Email;

        [global::ProtoBuf.ProtoMember(4, Name = @"phone")]
        public global::System.Collections.Generic.List<PhoneNumber> Phones { get; } = new global::System.Collections.Generic.List<PhoneNumber>();

        [global::ProtoBuf.ProtoMember(5, Name = @"test_packed", IsPacked = true)]
        public int[] TestPackeds { get; set; }

        [global::ProtoBuf.ProtoMember(6, Name = @"test_deprecated")]
        [global::System.Obsolete]
        public int TestDeprecated
        {
            get { return __pbn__TestDeprecated.GetValueOrDefault(); }
            set { __pbn__TestDeprecated = value; }
        }
        public bool ShouldSerializeTestDeprecated() => __pbn__TestDeprecated != null;
        public void ResetTestDeprecated() => __pbn__TestDeprecated = null;
        private int? __pbn__TestDeprecated;

        [global::ProtoBuf.ProtoMember(7, Name = @"foreach")]
        public int Foreach
        {
            get { return __pbn__Foreach.GetValueOrDefault(); }
            set { __pbn__Foreach = value; }
        }
        public bool ShouldSerializeForeach() => __pbn__Foreach != null;
        public void ResetForeach() => __pbn__Foreach = null;
        private int? __pbn__Foreach;

        [global::ProtoBuf.ProtoContract(Name = @"phone_number")]
        public partial class PhoneNumber : global::ProtoBuf.IExtensible
        {
            private global::ProtoBuf.IExtension __pbn__extensionData;
            global::ProtoBuf.IExtension global::ProtoBuf.IExtensible.GetExtensionObject(bool createIfMissing)
                => global::ProtoBuf.Extensible.GetExtensionObject(ref __pbn__extensionData, createIfMissing);

            [global::ProtoBuf.ProtoMember(1, Name = @"number", IsRequired = true)]
            public string Number { get; set; }

            [global::ProtoBuf.ProtoMember(2, Name = @"type")]
            [global::System.ComponentModel.DefaultValue(Person.PhoneType.Home)]
            public Person.PhoneType Type
            {
                get { return __pbn__Type ?? Person.PhoneType.Home; }
                set { __pbn__Type = value; }
            }
            public bool ShouldSerializeType() => __pbn__Type != null;
            public void ResetType() => __pbn__Type = null;
            private Person.PhoneType? __pbn__Type;

        }

        [global::ProtoBuf.ProtoContract(Name = @"phone_type")]
        public enum PhoneType
        {
            [global::ProtoBuf.ProtoEnum(Name = @"mobile")]
            Mobile = 0,
            [global::ProtoBuf.ProtoEnum(Name = @"home")]
            Home = 1,
            [global::ProtoBuf.ProtoEnum(Name = @"work")]
            Work = 2,
        }

    }

    [global::ProtoBuf.ProtoContract(Name = @"opaque_message_list")]
    public partial class OpaqueMessageList : global::ProtoBuf.IExtensible
    {
        private global::ProtoBuf.IExtension __pbn__extensionData;
        global::ProtoBuf.IExtension global::ProtoBuf.IExtensible.GetExtensionObject(bool createIfMissing)
            => global::ProtoBuf.Extensible.GetExtensionObject(ref __pbn__extensionData, createIfMissing);

        [global::ProtoBuf.ProtoMember(1, Name = @"messages_list")]
        public global::System.Collections.Generic.List<byte[]> MessagesLists { get; } = new global::System.Collections.Generic.List<byte[]>();

    }

}

#pragma warning restore CS1591, CS0612, CS3021, IDE1006

Woila, we have our csharp model. But wait, isn't this going to create the model class every time you build the project? Well... yes, it will and that is why we need to modify this to check if the model already exist not to create it.

There is one more disadvantage of this approach and that is that for each proto file you will have to add additional Exec node in the Target node. Not a big deal, but to me it is a code repetition, and we all know that is a big no no, so let's see how to fix these two problems.

For this purpose I used PowerShell script, but if you are not sunning Windows development machine, you can use PowerShell Core which I explained how to install on Linux environment in my post Remote PowerShell Core session to a Linux host from Windows machine. Basically, instead of our BeforeBuild call to protogen tool, we will call PowerShell script with all the parameters and do the file exists logic and protogen call inside it. I named the file protogen.ps1 and placed it right next to the tool, in the same folder

Param(
[Parameter(Mandatory=$true)]    
$protogenPath,
[Parameter(Mandatory=$true)]    
[string]$protoFolder,
[Parameter(Mandatory=$true)]    
[string]$modelFolder,
[Parameter(Mandatory=$true)]    
[string]$namespace
 )

$protoFiles = get-childitem $protoFolder -recurse -force -include *.proto   
foreach($protoFile in $protoFiles){

    $outFilePath = $modelFolder + "\" + $protoFile.BaseName + ".cs"

    if(!(Test-Path -Path $outFilePath)){

        $protoPathParam = "--proto_path=" + $protoFolder
        $csharpOutParam = "--csharp_out=" + $modelFolder
        $packageParam = "--package=" + $namespace

	    Start-Process -FilePath "dotnet" -ArgumentList $protogenPath , $protoFile.Name, $protoPathParam , $csharpOutParam , $packageParam
    }

}

We also need to modify our .csproj file a bit in order to call PowerShell script instead of calling protogen tool directly

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Folder Include="Models\" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="protobuf-net" Version="2.3.13" />
  </ItemGroup>

  <Target Name="protogen" BeforeTargets="BeforeBuild">
    <Exec Command="powershell -file $(SolutionDir)tools\protogen\protogen.ps1 -protogenPath $(SolutionDir)tools\protogen\protogen.dll -protoFolder $(ProjectDir)Protos -modelFolder $(ProjectDir)Models -namespace $(ProjectName).Models" />
  </Target>

</Project>

Now this is a lot better and you will get your model only if there is no C# class for the proto file in your project.

References

Disclaimer

Purpose of the code contained in snippets or available for download in this article is solely for learning and demo purposes. Author will not be held responsible for any failure or damages caused due to any other usage.

About the author

DEJAN STOJANOVIC

Dejan is a passionate Software Architect/Developer. He is highly experienced in .NET programming platform including ASP.NET MVC and WebApi. He likes working on new technologies and exciting challenging projects

CONNECT WITH DEJAN