Re: Assert failure on 'list_member_ptr(rel->joininfo, restrictinfo)'

Andrei Lepikhov писал(а) 2023-12-08 07:37:
> On 28/11/2023 01:37, Alexander Korotkov wrote:
>> On Mon, Nov 27, 2023 at 8:07 PM Andres Freund <andres@anarazel.de> 
>> wrote:
> Sorry for the late answer, I missed this thread because of vacation.
>>> On 2023-11-27 11:29:48 +0530, Ashutosh Bapat wrote:
>>>> How do we ensure that we are not making unnecessary copies of 
>>>> Bitmapsets?
>>> 
>>> We don't - but that's not specific to this patch. Bitmapsets 
>>> typically aren't
>>> very large, I doubt that it's a significant proportion of the memory
>>> usage. Adding refcounts or such would likely add more overhead than 
>>> it'd save,
>>> both in time and memory.
> 
> I'd already clashed with Tom on copying the required_relids field and 
> voluntarily made unnecessary copies in the project [1].
> And ... stuck into huge memory consumption. The reason was in 
> Bitmapsets:
> When we have 1E3-1E4 partitions and try to reparameterize a join, one 
> bitmapset field can have a size of about 1kB. Having bitmapset 
> referencing Relation with a large index value, we had a lot of (for 
> example, 1E4 * 1kB) copies on each reparametrization of such a field. 
> Alexander Pyhalov should remember that case.

Yes. If it matters, this happened during reparametrization when 2 
partitioned tables with 1000 partitions each were joined. Then 
asymmetric  pw join managed to eat lots of memory for bitmapsets (by 
lots of memory I mean all available on the test VM).

> [1] Asymmetric partition-wise JOIN
> https://www.postgresql.org/message-id/flat/CAOP8fzaVL_2SCJayLL9kj5pCA46PJOXXjuei6-3aFUV45j4LJQ%40mail.gmail.com

-- 
Best regards,
Alexander Pyhalov,
Postgres Professional