To obtain the vector defined by two points, where P1 is the origin and P2 is the end point:
P1 (P1x,P1y,P1z)
P2 (P2x,P2y,P2z)
V12=(P2x-P1x, P2y-P1y, P2z-P1z)
You can use this to calculate the vectors that the vertices of the triangle define (see below), or the vector defined by the origin and one of the vertices of the triangle.
The scalar product of two vectors A (A1,A2,A3) and B(B1, B2, B3) is what I wrote in my previous post:
A·B=(A1B1+A2B2+A3*B3)
Since the resulting value can also calculated as the length (or module) of the first vector multiplied by the module of the second vector and by the cosinus of the angle between the two vectors, the result will be positive for angles below 90 º and negative for angles above 90 º.
We need a way to calculate the vector normal/perpendicular to the plane defined by the triagle (which I thought you may have already because I have seen that vector included in mesh data) using the three points that define the triangle. If we look at the plane of the triangle perpendicularly and your points are ordered clockwise, I will consider that the vector coming towards us is positive (we need to establish a sign criterion).
Obtain the vector defined by points 1 and 2, using 1 as reference:
V2=(P2x-P1x, P2y-P1y, P2z-P1z)
Obtain the vector defined by points 1 and 3, using 1 as reference:
V3=(P3x-P1x, P3y-P1y, P3z-P1z)
To obtain the vector that is perpendicular to the triangle (and to V2 and V3) you need to calculate their vectorial product (as Ramon said above):
N = V3 x V2
N =(V3yV2z-V3zV2y, V2xV3z-V3xV2z, V3xV2y-V3yV2x)
Final step: calculate the scalar product of that N and the vector defined by the origin and a vertex of the triangle.
HTH,
Julen