ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation

[clebertsuconic]

No description provided.

[tabish121]

Overall looks good, nice simplification of some of the message handling

[clebertsuconic]

I added a commit to be squashed, returning the RELOAD state and persisting the memory estimate on the storage (journal or paging).

I honestly don't think the scan will make any difference.. The real issue is that the estimates were wrong in the first place...

But this commit should settle any discussion.

I think There are already version tests in the Version tests. but i will make sure about it soon.

[clebertsuconic]

for some reason I'm having to use a different persister, and I would have to digg on previous versions to understand why.

I will do some investigation tomorrow, but most likely I will need the V4 persister.

[clebertsuconic]

The reason I had to add a V4 is because of Paging:

On this part here, I encode the number of queues and the queueIDs:

artemis/artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/impl/PagedMessageImpl.java

Lines 227 to 234 in 5b12277

	int queueIDsSize = buffer.readInt();
	queueIDs = new long[queueIDsSize];
	for (int i = 0; i < queueIDsSize; i++) {
	queueIDs[i] = buffer.readLong();
	}

At the point of the encoder I don't have any reference to the number of bytes used by its own decoder.

I'm adding one on V4 now, if in the future anything else is added, we will stop reading at that marker.

[clebertsuconic]

I intend to squash the second commit on the first as soon some review is done on the new persister.

[clebertsuconic]

I'm setting it as ready to review. but this commit should be squashed before merged.

[clebertsuconic]

I needed to add versioning to AMQPLargeMessagePersister as well.

I could choose to skip memory calculations on large message... however I will prefer to keep it the same way as StandardMessage.

this is also better for future additions

[clebertsuconic]

I added a commit addressing the comments, let me know when I can squash it please

[clebertsuconic]

I added another commit to be squashed after review

[gemmellr]

I can see is that this would in future allows older brokers unaware of future changes to this new persister, to skip over anything added that they dont understand...but, do we actually send these 'persister encodings' over the wire at all? Wouldnt that make these changes already incompatible with existing versions if so? I kinda wonder if thats worth it if so, in order to write information thats already available via the message.

Either way, if we ever changed this v4 persister (or the v2 large) in future to do something else, and start writing a bigger value here so that old brokers can skip it, then surely we end up having to have the new broker inspect the size during decode and guess from any possible 'past variants of v4' what specific things were added 'without changing he persister [version]' are actually present or not in any given v4 (/v2 large) message encountered, purely from the size it finds?

That seems like it could be a bit of a PITA. Would it not be simpler long term to just have a basic list/map/array style structure so that we can always easily determine/look-up the values the given version understands and not need to be concerned with manually determining anything when additions are made? If we did that at the start then perhaps it would still be the V1 persister in use, unless we made some sort of drastic removal?

[clebertsuconic]

we already have extraProperties... but I don't want to use it here.. I wanted to keep it lean and just encoded

[gemmellr]

Is it expected there are now no combinations with a new main broker and old backup? There were 3 before which seems a bit of a switch up. EDIT: ah, so is that about persister incompatibility?

[clebertsuconic]

version compatibility support is alwasys forward . the backward compatibility is best effort.. and this case the journal persistence is broken by V4. the older version won't understand V4.