To create a boundary you need to include your boundary 3DPolyline in the triangulation as a breakline. Therefore yo need to check the box "Polyline (Vertex)" and select layer that contains the 3Dpolyline. Select all your items for the triangulation (boundary 3Dpolyline included) and triangulate.
You now will end up with a bunch of triangles probably inside and outside the boundary line. However if you plot your triangles to a new layer in the drawing (using Edit/View Triangles) and then copy your boundary line into that drawing you will see that triangles interfering your boundary line are cut at the location of the line.
Now you can use a tool in "Edit/View Surface" to delete triangles that are outside or inside a 3DPolyline (your boundary). If you have multiple boundaries you can do the same, include them in the triangulation as breaklines and the delete unwanted triangles manually. Finally you save all your remaining triangles as a surface using "Surface/Save Surface..". CadTools will recognize valid surface triangles while saving triangles from drawing to file, all other items will be excluded.
