To know how to build JAMin, see Section 3.12.
Masterisation is a complex step and it exists a lot of books and tutorial.
Here is the advices of one of the Ardour's developers on using JAMin with Ardour:
Open up either the editor mixer for the master bus or use the mixer window
Right click on the black box above the master fader to open the context menu
Select(it will automatically be a stereo insert)
"Edit" the insert either via right clicking on the insert and selecting, or "edit click" (CTRL-Right click) the insert
Connect JAMin inputs and outputs to the send outputs and inputs
"Activate" the insert by either right clicking on the insert and selectiongor middle-click on the insert