Filtering and paging collection result in ASP.NET Core Web API

Paging response data in ASP.NET Core WebApi projects

Paging a REST API response is equally important as filtering the data returned through REST API endpoints and they quite often come together. Just like filtering, paging limits the amount of data returned from the endpoint, saving both client and server side resources. Imagine if you would return all customers data for only one customer record or you return all of the filtered data when record you are searching for can be among first few records returned.

It would simply result in a waste of processing power and network bandwidth on the server, and a unnecessary complexity in the implementation of the client.

There are technologies that can solve these two problems such as OData or GraphQL. However, It's also possible to solve these problems with just custom solutions that rely on filtering and paging the API data. 

Why would you consider custom solutions instead of already existing technologies? Simply because it can avoid your team to waste time learning new technologies when they can already solve these problems without them. Moreover, these technologies would impose on the client to adapt and be restricted to their corresponding clients. Relying on filtering and paging techniques is also not free of complexity. Sometimes these simple approaches become so complex that they in the end become a problem for the consumers/clients.

I will mention some of the pillars of the custom filtering and paging in REST API design, specifically when using .NET Core WEB API for implementing filtering and paging in REST services.

Make filters easily extensible

One of the common mistakes I noticed in implementing custom filtering and paging in WEB API projects is passing values as separate parameters to MVC Controller Action method. Not that it is not only easy to extend, but you are making your method signature more complex with tendency to get even more complicated with adding more filtering options to the endpoint.

        [HttpGet]
        public IActionResult Get(String term, int page, int limit)
        {
            //Handle filtering and paging
        }
    

Let's imagine that after some time you have to extend your endpoint to receive DateTime and Boolean parameter which will be involved in filtering. The method signature will change and become:

        [HttpGet]
        public IActionResult Get(String term, DateTime minDate, Boolean includeInactive, int page, int limit)
        {
            //Handle filtering and paging
        }
    

You can already see that after one update your method signature becomes more complex and to make it readable, you should probably break it into multiple lines.

Unless you have versioning in place, it will be hard to be in-line with the existing endpoint consumers as well as they will not be aware of new parameters and they will automatically start getting 404 response as MVC will not know how to route their request and will fail to match the signature. This means you have to make them optional and move them at the end of the parameters list.

        [HttpGet]
        public IActionResult Get(String term, int page, int limit, DateTime? minDate = null, Boolean includeInactive=false)
        {
            //Handle filtering and paging
        }
    

Now parameters are scattered in the signature leaving your method signature with mixed filtering and paging parameters with no logical grouping or order because you cannot have optional parameters other than at the end of the parameters list in the method signature. 

The complexity blows up more when you realise that you have to apply these changes to more than one method. You might have to make changes to more than one endpoint at the same time which makes not only one method but pretty much the whole application facade harder to maintain.

I think we made enough points to conclude that using multiple parameters for the filtering method is a bad bad idea. Better way to do this is using model and put all parameters as properties of the POCO class. Although method is still HTTP GET, MVC can bind the model for you from the query string by using [FromQuery] keyword for the model.

        public class FilterModel
        {
            public String Term { get; set; }
            public DateTime MinDate { get; set; }
            public Boolean IncludeInactive { get; set; }
            public int Page { get; set; }
            public int Limit { get; set; }
        }

        [HttpGet]
        public IActionResult Get([FromQuery] FilterModel filter)
        {
            //Handle filtering and paging
        }
    

Extending the filter is now the responsibility of a single class and if your filter is generic across the project or it has common properties (like page number and page size/limit) you can take it out to the base class and if you need to extend the filter model across multiple Actions or even across multiple Controllers, you can just extend the base model filter class. You would still have to handle the new parameters in your filtering logic, but your method signatures will stay the same without extending it.

Don't let the client dot the job for you

Many of the custom implementations I saw let the client form the query string to get the next page. In my opinion this is not the right way to do it. One of the implementations I saw and I really like is the one ZenDesk API uses. Apart from the collection of entities, the response also consists of the URL to the next and the previous page of the result. A sample response would be something like this

{
	persons:[
		{
			name: "John Smith",
			dob: "1984-10-31",
			email: "john@smith.test.com"
		},
		...
	],
	nextPage: "http://localhost:5000/api/persons?name=John&page=2&limit=100",
	previousPage: null
}
    

This way your client does not have to take case of determining what is the next page URL and with today's modern infinite scroll on most of the UI implementations (both web and mobile) this approach works perfect as each page scroll to the bottom is a new HTTP GET to the next page URL.

In WEB API this would look something like this

        public class FilterModel
        {
            public String Term { get; set; }
            public DateTime MinDate { get; set; }
            public Boolean IncludeInactive { get; set; }
            public int Page { get; set; }
            public int Limit { get; set; }
        }

        public class PagedCollectionResponse<T> where T : class
        {
            public IEnumerable<T> Items { get; set; }
            public Uri NextPage { get; set; }
            public Uri PreviousPage { get; set; }
        }

        public class Person
        {
            public String Name { get; set; }
            public DateTime DOB { get; set; }
            public String Email { get; set; }
        }

        [HttpGet]
        public ActionResult<PagedCollectionResponse<Person>> Get([FromQuery] FilterModel filter)
        {
            //Handle filtering and paging
        }
    

This is just a shallow structure of how filter action signature should look like. Next, I will explain with a simple piece of code how to implement this approach in ASP.NET Core WEB API sample controller

Simple example of filtering and paging

To show you how to implement the above described paging approach with page pointer URL, I will use a simple controller and static collection of strings. Ideally you would query a repository for data, but to keep things simple and to focus on producing response structure as described, I'll stick to a simple collection of strings.

First thing before we jump to the logic will be to create models. Since any filtering in our project will receive page and limit value it makes sense to make this an abstract class so that any filter with paging can inherit it.

namespace Sample.Web.Api.Models
{
    public abstract class FilterModelBase:ICloneable
    {
        public int Page { get; set; }      
        public int Limit { get; set; }

        public FilterModelBase()
        {
            this.Page = 1;
            this.Limit = 100;
        }

        public abstract object Clone();
    }
}
    

We have a default constructor which sets the page size (Limit property) to 100, meaning by default, any filter model will page values in collections of 100 items. We also implement ICloneable interface but implementation is left as abstract to allow the inherited class to deal with the clone logic as it can involve additional properties of inherited POCO class. When we start implementing the paging logic, you will see why we need ICloneable interface involved.

Now let's have our filter implemented by inheriting FilterModelBase abstract class

    public class SampleFilterModel:FilterModelBase
    {
        public string Term { get; set; }

        public SampleFilterModel():base()
        {
            this.Limit = 3;
        }


        public override object Clone()
        {
            var jsonString = JsonConvert.SerializeObject(this);
            return JsonConvert.DeserializeObject(jsonString,this.GetType());
        }
    }
    

Apart from Page and Limit properties I added one additional property Term which should be used to filter our strings collection. I also set new page size in the constructor to 3 instead of default 100 assigned in the base class constructor as want to see paging on the small set of data. Clone method represents deep copy of filter model instance and it is done with simple serialization/deserialization using Newtonsoft.Json package.

With this we covered our Action method input, but now we need to take care of the output. To make response generic, I will use same structure model, but I will change the type of the collection as needed by the controller. For this purpose I used generic type for declaring the output model, so that we can re-use it in multiple Controller Action methods to return different types of collection elements.

namespace Sample.Web.Api.Models
{
    public class PagedCollectionResponse<T> where T:class
    {
        public IEnumerable<T> Items { get; set; }
        public Uri NextPage { get; set; }
        public Uri PreviousPage { get; set; }
    }
}
    

As we are going to return collection of persons mentioned above, we can use the same model class for storing our sample data.

namespace Sample.Web.Api.Models
{
    public class Person
    {
        public String Name { get; set; }
        public DateTime DOB { get; set; }
        public String Email { get; set; }
    }
}
    

We are ready now to write down our page handling. As I mentioned I will use collection of Person class instances and for this demo I declared them as a collection initiated at the Controller construction

namespace Sample.Web.Api.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class PersonsController : ControllerBase
    {

        IEnumerable<Person> persons = new List<Person>() {
            new Person() { Name = "Nancy Davolio", DOB = DateTime.Parse("1948-12-08"), Email = "nancy.davolio@test.com" },
            new Person() { Name = "Andrew Fuller", DOB = DateTime.Parse("1952-02-19"), Email = "andrew.fuller@test.com" },
            new Person() { Name = "Janet Leverling", DOB = DateTime.Parse("1963-08-30"), Email = "janet.leverling@test.com" },
            new Person() { Name = "Margaret Peacock", DOB = DateTime.Parse("1937-09-19"), Email = "margaret.peacock@test.com" },
            new Person() { Name = "Steven Buchanan", DOB = DateTime.Parse("1955-03-04"), Email = "steven.buchanan@test.com" },
            new Person() { Name = "Michael Suyama", DOB = DateTime.Parse("1963-07-02"), Email = "michael.suyama@test.com" },
            new Person() { Name = "Robert King", DOB = DateTime.Parse("1960-05-29"), Email = "robert.king@test.com" },
            new Person() { Name = "Laura Callahan", DOB = DateTime.Parse("1958-01-09"), Email = "laura.callahan@test.com" },
            new Person() { Name = "Anne Dodsworth", DOB = DateTime.Parse("1966-01-27"), Email = "anne.dodsworth@test.com" }
            };

        // GET api/values
        [HttpGet]
        public ActionResult<PagedCollectionResponse<Person>> Get([FromQuery] SampleFilterModel filter)
        {

            //Filtering logic
            Func<SampleFilterModel, IEnumerable<Person>> filterData = (filterModel) =>
            {
                return persons.Where(p => p.Name.StartsWith(filterModel.Term ?? String.Empty, StringComparison.InvariantCultureIgnoreCase))
                .Skip((filterModel.Page-1) * filter.Limit)
                .Take(filterModel.Limit);
            };

            //Get the data for the current page
            var result = new PagedCollectionResponse<Person>();
            result.Items = filterData(filter);

            //Get next page URL string
            SampleFilterModel nextFilter = filter.Clone() as SampleFilterModel;
            nextFilter.Page += 1;
            String nextUrl = filterData(nextFilter).Count() <= 0 ? null : this.Url.Action("Get", null, nextFilter, Request.Scheme);

            //Get previous page URL string
            SampleFilterModel previousFilter = filter.Clone() as SampleFilterModel;
            previousFilter.Page -= 1;
            String previousUrl = previousFilter.Page <= 0 ? null : this.Url.Action("Get", null, previousFilter, Request.Scheme);

            result.NextPage = !String.IsNullOrWhiteSpace(nextUrl) ? new Uri(nextUrl) : null;
            result.PreviousPage = !String.IsNullOrWhiteSpace(previousUrl) ? new Uri(previousUrl) : null;

            return result;

        }
    }
}

    

The sample code is pretty raw and it is not production, it requires some polishing to be re-used in multiple controllers, but it serves the purpose of producing the desired output data structure and paging logic on the small sample of simple data collection.

Let's go step by step and analyze the logic blocks of the method

  • Filtering logic
    Simple Func that returns collection of items from the source data collection takes care of fetching batches of objects based on the passed filter model. This implementation pretty much depends on your filtering logic and your data you are applying the filter. The Func body is Action method specific.
  • Get the data for the current page
    A simple usage of the above mentioned Func implementation. The reason for putting the logic inside the Func is for later re-use to determine next and previous page URLs.
  • Get the next page URL string
    Here, we are creating a new model with an updated page number. Remember we used ICloneable for the filter POCO? Now we are going to use it to create deep copies and update the page number of the model so we can generate the URL for the next page.
    Before we generate the URL of the next page we need to know if there are any elements returned for the next page number. We don't want to send our client to an empty collection response in the next page as we want our client to rely solely on NextPage and PreviousPage URL properties.
  • Get previous page URL string
    Pretty similar to getting NextPage URL with a slight difference in logic. We do not need to count the result set collection as for the next page URL. We only need to check if the page number is 1 This means there are no more pages and PreviousPage URL is null value.

We have pretty much everything covered, so let's see how this actually works in POSTMAN.

Api Paging 1

In initial request with default paging parameters we can see that we have 3 Persons in the result collection, our NextPage URL points to URL which have page number increased by one and our PreviousPage URL is null as we are on the first page and there are no pages before it.

If we follow NextPage URL and execute HTTP GET of it in POSTMAN we'll get the following response.

Api Paging 2

You can see now that we have both NextPage URL as well as PreviousPage URL. If you noticed we have 9 elements in our sample data collection, which means request to the NextPage URL should give us 3 more elements in the result collection.

Api Paging 3

Our last page returns 3 Persons in the result collection, but you can notice that NextPage URL is null. That is because count for the page number 4 will return no elements in the response and we are notifying the consumer that we do not have any more data to return.

I demonstrated this WEB API response paging on a simple data collection, but real-life scenario would involve data filtering and querying data repository. I hope in that in the near future I will be able to write more detailed text with more polished implementation with a showcase of using Repository Pattern and re-usable logic which can be applied to multiple controllers and actions without any code repetition.

References

Disclaimer

Purpose of the code contained in snippets or available for download in this article is solely for learning and demo purposes. Author will not be held responsible for any failure or damages caused due to any other usage.


About the author

DEJAN STOJANOVIC

Dejan is a passionate Software Architect/Developer. He is highly experienced in .NET programming platform including ASP.NET MVC and WebApi. He likes working on new technologies and exciting challenging projects

CONNECT WITH DEJAN  Loginlinkedin Logintwitter Logingoogleplus Logingoogleplus

JavaScript

read more

SQL/T-SQL

read more

Umbraco CMS

read more

PowerShell

read more

Comments for this article