Response probability density of a system with cross-correlated parametric and additive input noises

Data processing and subsequent mining is a widely followed task. Employment of suitable evaluation and interpretation procedures can signiﬁcantly improve the effective resolution of measuring facility using an identical hardware equipment. Recording of time variable processes is accompanied by various internal disturbing effects as a rule. They inﬂuence parameters of the measuring facility, transducer-device transmitting, etc. These parasitic processes are usually of the random character and, consequently, they exercise as parametric noises. Moreover, the input signal mostly consists of a useful signal, which can be taken for deterministic, and of a random additive part. Due to interaction of additive noises with the device itself, the cross-correlation of both additive and multiplicative noises cannot be neglected as a rule. Various combinations of noises are the origin of random and also systematic measuring errors which can have under certain circumstances a cumulative character. Their inﬂuence deteriorates the output signal quality and can lead ﬁnally to the stochastic stability loss. These effects can be theoretically described using differential systems with stochastic coefﬁcients and a stochastic right hand side considering all input and output processes to be of the Markov type. A direct investigation of the relevant Fokker-Planck equation is employed as the main tool. Two ﬁrst stochastic moments (mathematical mean value and variance) as evolutionary processes are investigated for a general deterministic useful signal and subsequently for two special cases of this one. Both types of input random noises are considered. Conditions of stochastic stability with respect to intensities of input random processes are formulated. The probability density function is deduced as well, in order to illustrate the probabilistic character of the system response as a whole. The stochastic asymmetry of the output signal is identiﬁed. Limitation procedures show a smooth transition from a general stochastic problem to deterministic noise free input signal and its processing.


Introduction
Measurement of dynamic quantities is usually accompanied by various interfering processes, see Fig. 1.They affect the useful signal f (t) on its way from a sensor element to a measuring device.This disturbing process ϕ(t) is mostly random and is added to the useful signal which is tracked.Therefore, on the input of a measuring device the sum of both is perceived.Hence, a subsequent filtering through a device being given by an operator D{•} works with the process f (t)+ϕ(t).Inside of a device some more inevitable perturbations w(t) are generated influencing parameters of the device itself.Therefore, such perturbations have a parametric (multiplicative) character and they are random as well.The output signal from the device u(f, w, ϕ), which is then recorded for further data processing, is a function of the useful signal and additive and Fig. 1.The signal flow in a measuring facility: (a) recording equipment, (b) measuring device, (c) sensor element; D{•} is the operator transforming input into output signal parametric random noises.For the theoretical background of random processes, their filtering and other manipulation, see well known monographs [4,8,13] and others.
Different combinations of additive and parametric noises are an origin of not only random but also systematic errors of measurement, which may be cumulative in certain circumstances and can result in degradation of the recorded signal; in particular its resolution ability and information reliability.The system may become stochastically unstable due to random noises, see, e.g., [8] or [10].Influencing the mathematical mean value and shape of the response probability density due to parametric random noises is a general phenomenon and occurs in all types of linear and non-linear operators.
It is worthy to recognize character of such perturbation and try to filter out their influence onto the output processes.Such an intervention can significantly improve resolution of measurements using the same equipment.At the same time, it is necessary to be aware that the random part of output processes is non-Gaussian in general, even if all input processes are Gaussian (commonly adopted assumption).There are several reasons for that: parametric noises and their possible interaction with additive noises, possible nonlinearity of the signal processing in the measuring facility, etc.Therefore, an employment of conventional correlation and spectral methods becomes problematic and formulation based on the theory of Markov processes and "associated" Fokker-Planck Equation (FPE) is necessary.

Mathematical model
For the sake of clarity, we choose the simplest conceivable model of the measuring system which can be considered; a simple first-order tracking system with a feedback.The noises will be considered as the Gaussian centered processes of known characteristics.The behavior of the system under consideration can be described by a linear stochastic differential equation (Langevin equation):
Both noises w(t) and ϕ(t) are narrowly related, as the additive noise influences significantly the functionality of the device itself and, consequently, the processing of the signal itself.It follows from numerous experiences in measuring practice.Therefore, the cross-intensity s wϕ should be taken into account.Indeed, even the original theory of the optimal filtering defines and works with the full correlation (or intensity) matrix of input noises whatever they are of additive or multiplicative type, see monographs [2,4,13] and many more.
The system modeled by Eq. ( 1) is intended to track the useful signal f (t), so that its ideal output is: where a 0 is a constant converting one physical quantity into another one.However, the actual signal u(t) as a rule is different from the ideal signal u f (t).The ideal signal u f (t) cannot be reached directly and we can ask only a certain optimum, which can be deduced from a minimal variance of difference of both signals: where: u d (t) -mathematical mean value of the process u(t), D u (t) -variance of the process u(t), E {•} -operator of the mathematical mean value with respect to Gaussian probability density.Take a note, that the Gaussian closure of the output signal is assumed to enable applicability of the operator in this meaning.
To solve the problem of the optimal approximation of the ideal signal u f (t), in principal, two strategies can be employed.The first group of procedures is based on the indirect reconstruction of the output signal; it is known as the optimal or sub-optimal filtering, see, e.g., [2,6,13].
An indirect way consists in evaluation of a certain Bayesian conditional probability of the response, while Eq. ( 3) is not used as a primary criterium.This procedure is apparently well substantiated with a strong mathematical background.It leads to a substantial generalization of the FPE, which includes integral and nonlinear terms.So, the generalized FPE is very complex and any determination of parameters, even those of a simpler sub-optimal filter, requires further simplifications in order to get through.Moreover, a filter following from the generalized FPE is mostly of a multi-pass type.In such a case, as a rule, these filters cannot be used on-line at the output of the measuring device and employed, for instance, for a feedback control.So that, the practical application of the filter is complicated and inflexible.
On the other hand, the indirect way is always more robust, even if incomparably more complicated in solution of the generalized FPE and mainly in application of the resulting filter.In general, indirect procedures based on the generalized FPE analysis are preferable in conditions when disturbances are large or when a high degree of "polishing" is necessary.
The second group of procedures is based on the direct investigation of a stochastic differential system with combined deterministic and random input modeling behavior of a measuring system as a whole.There are many special cases where this approach is eligible and, hence, it is preferrable.Practical aspects of this approach have been discussed in a number of papers, e.g., [14].
There exists a significant overlap of both groups, but basically the direct procedures reveal to be effective in simpler cases.In principle, this style of data processing has been brought into consideration in the early nineties, see, e.g., [12] and others.A large variety of studies followed, addressing many special domains of mechanics and informatics, [5,9] and other disciplines, e.g., [7].These studies strive to extend deterministic methods into employment in stochastic domain or to embed their background into conventional correlation or spectral approaches.As a rule, many of them do not involve the basis of Markov processes and FPE, although also here exceptions should be referred to, e.g., [3].
We try to keep the style of the direct group approaches, but we deeply involve the tool based on Markov processes and relevant FPE.The reason is that the conventional FPE enables to develop a very flexible variety of procedures respecting a strong mathematical background and, therefore, keeping a position being free of subjective hypotheses which are inevitably adopted outside this strategy.
Basically, u d (t), D u (t) and corresponding p(u) are investigated.However, it can be shown, see [13], that the minimal variance σ 2 , see (3), can be reached using a direct way by adopting u d (t) instead of ideal output u f (t), which is inaccessible.As the direct way, it is considered an analysis of the relevant FPE producing an adequate number of the first stochastic moments of u(t) (here the 1st and 2nd moments are assumed to be convergent in the meaning of the Gaussian closure).This treatment follows, in general, from properties of the Ito stochastic differential which results in a classical version of the FPE.Of course, a number of assumptions concerning the input processes and the system parameters should be fulfilled.
Therefore, the reconstruction in fact consists in evaluation of u d (t), which provides the minimal variance σ 2 when replacing the original signal by the stochastic moment series.Justifiability and reliability of this step or quality of u d (t) as a representant of the true signal follow from the subsequent evaluation of D u and mainly from the corresponding probability density p(u).Therefore, no explicit minimization of σ 2 should be done.
Let us revisit Eq. ( 1).The stochastic differential equation can be solved using one of the commonly used methods, see monographs [2,8,13] and others.With respect to the fact that the random parts of the input and output processes can be considered of the Markov type and, moreover, both input processes are Gaussian and centered, it is worthy to employ the FPE to search the unknown probability density function (PDF) p(u, t).
According to Eq. ( 1), we substitute into the formulas for the drift and diffusion coefficients, see, e.g., [8,13].After some manipulation, we get the FPE for the given problem: Based on the FPE, it is also easy to deduce the approximate equations for computation u d (t), K uu (t) = D u (t) which are the first two stochastic moments characterizing approximately the PDF of the response: where u d0 , D u0 are initial values of u d (t) and D u (t), respectively.The first two moments represent a qualitative approximation of the resulting PDF.It means in principle that the PDF of the response does not qualitatively differ from the Gaussian curve, which can be resolved into an asymptotically convergent series with respect to stochastic moments with Gaussian closure.
In particular, the output PDF should be uni-modal, integrable and smooth enough.In such a case, the first two moments are acceptable to estimate qualitatively behavior of the response.Eq. (5a) does not include D u (t) and, hence, it can be solved independently.We obtain: The parameter a 1 should be positive, otherwise stability of Eq. ( 6) would be lost.It implies: For t → ∞, the first term vanishes and it holds: Similarly like before, also here should be a 1 > 0 and, therefore, C > s ww /2.It is obvious that a systematic error −s wϕ /2a 1 emerges whatever is the useful signal, provided the cross-correlation of both input signals is non-trivial.We can see that the variance D u (t) of the output signal is not directly dependent on the useful signal.On the other hand, an influence of f (t) penetrates indirectly due to interaction of D u (t) with u d (t).

Mean value and variance of the response for constant and harmonic useful signals
Let us examine properties of the mathematical mean value of u d (t), or of the deterministic part of the response, see, for example [1,10].The first term in Eq. ( 6) is obviously asymptotically stable in the first moment u d (t), if Eq. ( 7) holds.Due to the fact that this term represents influence of the initial condition of the homogeneous equation, it determines the stability of the system itself.Provided the second term is estimated from above using the Schwarz inequality, we get again Eq. ( 7) on the one hand and on the other hand the condition of f (τ ) square-integrability in the interval (0, t).Therefore, Eq. ( 7) represents the condition of Eq. ( 1) stability in probability.The same result can be received when using the stochastic variant of the Lyapunov function, see, e.g., [2,4] or [10].
Let us consider two special cases: (a) The useful signal is constant: The mathematical mean value u c (t) with respect to Eq. (3) reads: In the stationary state, i.e. when t → ∞, and supposing validity of Eq. ( 7), it arises from Eq. ( 9): where u cs is the stationary value of u c (t), which is reached for t → ∞.Provided the system is free of noises, then obviously the ideal value of a 0 in meaning of Eq. ( 2) reveals C −1 .Putting that into Eq.( 11), one obtains for the systematic error when noises are respected: Even if the useful signal is trivial, for non-zero cross-correlation of input noises the mathematical mean value of the deviation Eq. ( 12) is different from zero.On the other hand, the systematic error is non-trivial for f 0 > 0 even for independent noises.In order to avoid these unwanted effects, it would be preferable to use a 0 following this formula: Fig. 2. Evolution of the output signal mean value u d (t) for a constant useful signal with amplitudes f 0 = 0.0, 0.2, 0.4, 0.6, 0.8 and various settings of parametric noise variance s ww and cross correlation of both noises s wϕ Parameter a 0 following Eq.( 13) can be interpreted as a transforming value introduced in Eq. ( 2).Its explicit evaluation is meaningful in the case of a constant useful signal, where the closed form solution of the problem is possible.Nevertheless, this assignment can be used, for instance, for calibration purposes.Some demonstrations of the output mean value u d (t) are obvious from Fig. 2. The systematic error is well visible in the first column of plots for f 0 = 0, but also for other f 0 is the horizontal asymptote (dashed line) visibly shifted due to the non-zero cross-correlation s wϕ .This shift disappears for s wϕ = 0, see the right column.
(b) The useful signal is harmonic: f (t) = f 0 sin ωt, u c (t) -formally coincides with u d (t), where Regarding u c in the meaning of Eq. ( 3), it is formally identical with u d .The only difference is in the phase shift ψ, which reads The mathematical mean value of the response or of the deviation in stationary state is harmonic with the same frequency as the input signal.The response is phase shifted, where this shift approaches −π/2 for a 1 → 0 and grows with the frequency of the useful signal.The mean value is non-zero and rises with increasing correlation of both noises and when s ww → 2C.The transition process prolongs with an increase of s ww , s wϕ and f 0 and shortens for increasing ω.Parameter a 0 affects only the phase shift.It can be noted that the formula characterizing the 3. Evolution of the output signal mean value u d (t) for a harmonic useful signal, frequency ω = 6.283, with amplitudes f 0 = 0.0, 0.4, 0.8 and various settings of parametric noise variance s ww and cross correlation of both noises s wϕ output influenced by random noises is simpler and more transparent for the harmonic useful signal than that for f (t) = const.The systematic error is the same when t → ∞ for both cases as we already found for the general case of the useful signal.The centered parametric noise does not influence the mathematical mean value of the response, unless it is correlated with the additive noise.In both cases it affects, however, the sensitivity of the system, which increases with intensity of the noise as far as the stability loss of the system.Let us become aware that the mean value u d (t) does not depend on intensity of the additive noise s ϕϕ , as it follows from Eq. ( 6) or (8).The additive noise applies indirectly together with the parametric noise through the cross-correlation of both noises.A few particular cases can be followed in Fig. 3. Keeping frequency ω = 6.283 of the mean value u d (t) is obvious as well as its phase shift.It applies also to the systematic error in the left column.In the right column, which corresponds to s wϕ = 0, the systematic error does not appear.
Let us revisit Eq. (5b) which will be used now to deduce the response variance.The solution of Eq. (5b) can be written for a general history of u d (t) in the form: Regarding the stochastic stability of the second moment we obtain the simple condition: which is more severe than the condition Eq. ( 7).This is in line with the general principle that the stability of higher moments can only be ensured by fulfilling stricter conditions, see, e.g., [2,10].At the same time, it can be said that the stability of the second moment (here variance) implicates the stability of the mathematical mean value, which does not hold reciprocally.The same commentary regarding stability of the integral member in Eq. ( 16) can be expressed as it was given in case Eq. ( 8), that is the necessary square-integrability of u d (t) at the interval (0, t).However, it is always satisfied if u d (t) is asymptotically stable.We use formula Eq. ( 16) for the both special input cases discussed above: (a) The useful signal is constant.We substitute Eq. ( 9) into Eq.( 16).After some manipulations we receive: For t = 0, the variance is equal to its initial value D u0 .The transition process consists of two parts.With respect to the ratio of a 1 and a 2 , the first member in Eq. ( 18) drops faster than the second.The curve approaches a horizontal asymptote: Provided it holds: The curve rises first, reaches the maximum, for example at D u0 = 0 in point: and then it descends to the asymptote Eq. ( 20).If the conditions Eqs. (21) are not fulfilled, the curve Eq. ( 20) approaches the horizontal asymptote from below.
The variance in a stationary state following Eq.( 20) is influenced by the useful signal only if the cross-correlation of both noises does not vanish.Otherwise (s wϕ = 0): With zero cross-correlation of parametric and additive noises (s wϕ = 0), we get to a classic formula, from which it follows that the variance decreases with shrinking both parametric and additive noises.An attempt for annulation of the numerator in Eq. ( 20) obviously does not seem to make sense.Considering that s 2 wϕ ≤ s ww • s ϕϕ , it would mean that a useful signal meets the condition: which is not realistic assuming a significant dominance of f 0 over ϕ(t).Some tendencies of stationary state reaching have been outlined in Fig. 4. Two types of horizontal asymptotes approaching with respect to noise intensities and cross-correlation can be followed, as well as the successive escape of the maximum to t → ∞ for rising f 0 , as it follows from Eq. ( 24).This process is also accelerated with the increasing cross-correlation s wϕ .Nevertheless, the basic qualitative separation limit is given by criterion C = 1.5s ww related with Eq. (24) as well.14) into Eq.( 16), one obtains after several modifications: 25) is similar to Eq. ( 18), which follows from the basic formula Eq. ( 16).With disappearance of the first term which diminishes faster, the dependence of the variance on the initial value vanishes and extinction of the second term eliminates any dependence on the initial mathematical mean value of the response.
In the stationary state, the variance remains characterized by a non-symmetric sinusoid with a delayed argument.In the case of uncorrelated noises, the dispersion approaches constant for t → ∞.It remains in force also in the case, when c 3 grows provided the value of s ww increases in admissible limits.Similar discussion concerning the existence of local maxima of curves D u (t) can be performed like in the previous case, only different parameters are used.
Notice that unlike the mean value, properties of the variance D u (t) strongly depend on the intensity of the additive noise; this regards namely duration of the transition process and the asymptotic value, see Eqs. ( 18) and (25).
Let us summarize that the applicability of the theoretical background is limited by conditions of stability of individual formulae, e.g., ( 6), ( 7), ( 17), (21), etc.However, we should be aware that a practical application can show that the reasonable limits are lower.It strongly depends on "dynamics" of the useful signal.The treatment of "dramatically" variable signals will be more sensitive to a possible stability loss.On the other hand, the indirect procedures of the optimal filtering suffer from this shortcoming too.

Probability density of the response
Let us examine the response PDF employing Eq. ( 4).We consider the stationary solution limiting t → ∞.If this limit exists, it implies that ∂p(u, t)/∂t = 0.The useful signal is taken constant: f (t) = f 0 = const.The equation passes into an ordinary differential equation that can be formally integrated: The PDF should vanish for u → ±∞ together with all derivatives.This can only be achieved if const.= 0.It means that Eq. ( 26) can be rewritten after the separation the variables as follows: which implies: where N is a normalization constant.The result of integration depends on the character of roots of the following equation: It holds that Δ = s 2 wϕ − s ww s ϕϕ ≤ 0, and, therefore, two cases can occur depending on the discriminant Δ: (a) partial correlation of noises: Δ = s 2 wϕ − s ww s ϕϕ < 0, a 1 = C − 1 2 s ww , a 2 = C − s ww .The roots are complex conjugate, so that the integral Eq. ( 28) has the form as follows: (b) full correlation of noises: Δ = 0. Eq. (29) has one double root u 1,2 = s wϕ /s ww and the integral Eq. (28) reads: Eqs. ( 30) and (31) show that the PDF p(u) is a non-symmetric curve for non-zero crosscorrelation of both input noises (s w,ϕ = 0), or if the parametric noise and the useful signal are non-trivial.The PDF following from Eqs. (30) and (31) and substituted into Eq.(28) has the form of a product which consists of a reciprocal value of a quadratic polynomial and an exponential with an odd argument (s w,ϕ ≥ 0).The first part describes the curve symmetrical with respect to the mean value, while the exponential part causes the actual eccentricity.The curve drops faster for u descending below the mathematical mean value and slower for u rising above the mean value.The derivatives of Eqs.(30), (31) show easily that the stationary points of PDF lie in u → ±∞ and in the zero point of the numerator of the integrand in Eq. (28): It reconfirms an asymmetry of the exponent q(u) and, consequently, also of the response PDF, if both cross-correlated parametric and additive noises operate in the system.With respect to the above discussion of the exponent q(u), we revisit the formula of the response PDF, Eq. ( 28), which reads: p(u) = N exp −q(u) .In general, taking into account, that various variants of q(u) exist depending on the correlation level of the input noises, the PDF can be evaluated.Take a note that except the general formula for p(u), three special cases are worthy to be outlined.They follow from the general formula by means of adequate limitations: (a) The cross-correlation of noises vanishes: s wϕ = 0.
The curve is non-symmetric due to the exponential part.The ratio of its influence is determined by the useful signal level.If the useful signal vanishes, the exponential part of Eq. ( 33) equals one and only the first part in absolute value remains in force.This part is symmetric with the maximum in point u = 0.The additive noise influences the variance only, but the type of the curve and position of its maximum in the origin is not affected.Influence of both random noises on PDF of the response is evident from Fig. 5.The asymmetry of the PDF increases for a rising level of the useful signal f 0 .For f 0 = 0, the PDF is symmetric, but it does not match the Gaussian curve.It has higher variance and functional values especially for higher values |u|.The probability density is sharper at lower additive noise level.
(b) The parametric noise is not present: s ww = s wϕ = 0. Limiting Eq. (30) for s ww → 0, s wϕ → 0, one obtains: which is the Gaussian curve offset by f 0 /C outside the basic position.This shift is given by the nominal value of the input signal and by the nominal value of the system transmission parameter.Therefore, it is completely of the deterministic origin.The additive noise ϕ(t) determines the variance of deflection.It can be seen that only the additive Gaussian noise leads to the genuine Gaussian stochastic part of the system response, as it can also be seen from Fig. 6b.The shift of its peak in the positive direction is given only by the level of the useful signal.The shape of the curve does not change.As a comparison, Fig. 6a demonstrates the situation where the additive noise completely disappears and only the parametric noise is acting, see Eq. (33) for s ϕϕ .For increasing values of the useful signal, the PDF is heading for to a strong asymmetry.It is the result of influence of the exponential, the argument of which changes strongly in a neighborhood of u = 0 from positive to negative values due to arctg function.This effect is rising with increasing f 0 .
(c) The both noises disappear: s ww = s wϕ = s ϕϕ = 0. Limiting formula for p(u) in Eq. (34) for s ϕϕ → 0, one obtains: The task degenerates into a deterministic problem.The PDF changes into the Dirac function with a non-zero value at the peak of the Gaussian curve in the previous case.In other words, the deterministic response of the system occurs with probability one in this point and a zero probability anywhere else.

Conclusion
The procedure of reconstruction of the perturbed measuring signal is proposed.The reconstruction is defined in the meaning of the minimal variance of an ideal output and its mathematical mean value of the signal recorded.The formulation in terms of Markov processes together with the relevant Fokker-Planck equation is presented.Besides the general case, two particular cases of the useful signal (constant, harmonic) in the form of the first and second stochastic moments (mathematical mean value, variance) are discussed.Furthermore, the direct solution of the Fokker-Planck equation is investigated as an important complementary information.
When the response of a simple linear system with random noises in parameters and additive noises in the input signal is analyzed, a number of consequences that cannot be observed in purely deterministic tasks appears.If the correlation of parametric and additive noises is non-zero, the response PDF is asymmetric even if the useful signal vanishes.The response has a non-zero mathematical mean value although the both noises are centered processes.A similar effect can also be observed for a deterministic problem coefficient of which includes a harmonically variable component of the same frequency as the external load.In the case of a stochastic problem, this relationship follows from correlation between parametric and additive noises.These effects produce the systematic error in the mathematical mean value of the response.This must be thoroughly considered, if, for instance, the signal serves as an information which is subsequently integrated.Ignoring this effect would mean committing gross errors.The stochastic response asymmetry is, therefore, conditioned by a non-zero correlation of both noises and, of course, by existence of the parametric noise.An additive noise itself does not lead to the response asymmetry of the PDF.The random noises in parameters are typical for losing stochastic stability if the intensity of the noises exceeds a certain limit.However, it is a matter of the stability definition applied, what type of the stability loss occurs and how it physically manifests or how the system will behave in a sudden impact.
An indirect goal of the paper was to show that a relatively simple tool is able to provide useful results, if the problem is formulated in terms of Markov processes and the relevant Fokker-Planck equation.This strategy is based on mathematical steps free of any hypotheses.Limits of its applicability can be easily evaluated using formulae of the variance presented in the paper.Moreover, the authors are aware of a big potential of further development of this approach.It implies possible steps for the future, which consist in transformation of the whole procedure into a multi-dimensional form.This gives a possibility to work with much more complicated MDOF systems as mathematical models of a measuring facility (nonlinear, non-smooth coefficients in L 2 functional space, etc.).However, it would be necessary to replace a number of particular mathematical operations by more general formulations.Powerfulness of the optimal filtering is not called into question, but the relative simplicity enables the proposed procedure to serve as a more flexible tool in measuring data processing.

Fig. 4 .
Fig. 4. Evolution of the variance D u (t) for a constant useful signal and two settings of noise intensities