So if you just use ActivityStreams' default stuff, it's easier, because you have just a smaller set of vocab to deal with. Suddenly you want or encounter extensions and oh shit, you hit the -LD side of JSON-LD, and that stuff is there for a reason
The main thing you can and should do: compact incoming json-ld to your local world knowledge with your local json-ld context. (This is a one-line operation with any json-ld library.) Then you can treat it as "just json" and all the fields will be things you know. Store the fields you don't know about somewhere so you can get to them if you need them but they can be off to the side a bit.
@cwebber yeah that is exactly what I will be doing (when I will get to the point of doing the client part)
@cwebber fuuuck this reminds me i need to write a compactor