Pagination of nested relationships

[maryokhin]

Hello, recently started playing with this package, and while I am a bit struggling with somewhat scarce documentation, I really love the idea and how it works.

One of the quirks I have is how to paginate nested relationships. My data can easily reach thousands of relations per object and that sometimes just hangs the server/browser.

I read the spec and found out that it is possible to have the server paginate also the internal relationships.

A relationship object that represents a to-many relationship MAY also contain pagination links under the links member.

I would guess that this is not implemented yet? If not, can you evaluate on how hard it would be to do something like this and maybe point me in the right direction on where to start?

[maryokhin]

Here is what I found regarding implementation: http://stackoverflow.com/questions/15617595/paginate-relationship-in-django-rest-framework.

[jerel]

Based on that stackoverflow the first thing I would suggest trying is SerializerMethodResourceRelatedField. It allows you to declare a method on the serializer who's data will show up in the relationships json. I don't believe it currently works to return a queryset of many items (pull request #151 is meant to address that) so that PR might need to be cleaned up and merged first.

[maryokhin]

You mean an in-house implementation? I was thinking of maybe a generic implementation for the package. What I had in mind was:

• Having settings like DEFAULT_RELATIONSHIP_PAGINATION_CLASS, RELATIONSHIP_PAGE_SIZE etc. that are turned off by default.
• If setting is set, then paginate the relationships under the hood, serialize the links and so on.

So I was think how hard it would be to achieve something like this and how you look at that type of functionality being in the package. It just looks like functionality that anyone who has more than 20 relations per object would probably want.

[jerel]

I meant in-house implementation in case you needed it right now. We would like to see a proper solution in this package so if you feel to contribute it would be most welcome. @jsenecal and I had discussed the setting approach and if I remember right we decided it would work well but haven't had time to implement yet.

It seems to me like pagination styles should be uniform across the API so perhaps the main pagination class (rest_framework_json_api.pagination.PageNumberPagination) could be used by default with DEFAULT_RELATIONSHIP_PAGINATION_CLASS being optional? I haven't thought through the technical side of it, might not be possible

[maryokhin]

Should be possible if it doesn't break everyones API structure as soon as they upgrade, will take a look.

Btw, can you explain why the master branch is so behind develop? Because I'm using latest Django 1.9 & DRF 3.3.2 and can only use the develop branch of this package for my project.

[jerel]

This adapter was originally written for Ember Data and then turned into JSON API spec (so did Ember) so the master branch is there for legacy reasons. Very soon I plan to tag a beta version and we can keep a changelog/upgrade guide from then on.

[maryokhin]

Ah, ok, was just confusing, since it's usually the other way around, the legacy/snapshot code is in a different branch and master is the latest stable.

[maryokhin]

So I looked at possible ways to implement this. Because we obviously can't do it on the view level there are only 2 places this can be done: the serializer field responsible for the relations or directly in the renderer.

The 1st approach is much cleaner, but is not passible due to HyperlinkedIdentityField being inflated in the renderer (and we don't have control over that field). So this approach will be possible only if we expect/force everybody to extend from the custom ModelSerializer, meaning that shifts control of the serializer fields to the library developers and allows to implement much more on the field itself. Because if I can expect a ResourceRelatedFeild for every relation I can create a custom field that wraps the relation when ResourceRelatedFeild(many=True) is used by overriding many_init and that knows how/if it should paginate itself.

Looks like it boils down to the decision on where to put the overhead, on the users having to extend from a custom serializer, or the package developers implementing pretty much all the logic in renderer..

[jerel]

I expect everyone to import serializers from this package rather than directly from rest_framework

[maryokhin]

So I was off on a Christmas vacation for a while, but will try to take a fresh look at this now :)

[leifurhauks]

I don't know if it would be useful for this issue or not, but there is a rest_framework_json_api.serializers.ResourceIdentifierObjectSerializer that works a bit like the ResourceRelatedField but it's a full serializer. It's more generic though -- to use it to represent a relationship you'd want to instantiate it with the queryset of a related manager and many=True.

[Anton-Shutik]

Heys guys,
According to json api docs we can skip "data" in relationship at all, and just use "links" when fetching related entities.
I've created a PR that implements it.
Would love to hear your thoughts about it