pmf/pdf tables: http://gubner.ece.wisc.edu/ece730/tablesFromBookCovers2.pdf

See Course Guide information for this course.

=================================================

Wed. 1/23/2019: Introduction to the class. Review of some basic results from probability.

HW #1: Do problems 1 and 2 in ECE830notes2019_01_23.pdf.

Reading: In the notes, read everything except Section 1.2.2, which is not finished. Make a note (on paper - write it down) of any parts you don't understand so that we can discuss them in class on Friday.

=================================================

Fri. 1/25/2019: Refer to the updated notes: ECE830notes2019_01_25.pdf. Discussed simulation of Gaussian random vectors (Sec. 1.2.2). Started Chapter 2 and covered most of Sections 2.1 and 2.2 on estimation of random variables using the mean-squared error (MSE) criterion. Showed that the minimum MSE (MMSE) estimator is given by the conditional expectation as defined in Chapter 1. The key tool was the Orthogonality Principle.

Reading: In the updated notes, read Section 1.2.2 and Sections 2.1 and 2.2, which covers what we did in class today plus a bit more. Be sure to read Example 2.2 and the first part of the proof of the OP, which we did not cover in class, but you should be familiar with.

HW #2: In the updated notes, do Problems 3 and 4 in Chapter 1 and Problems 1 and 2 in Chapter 2. You will need GvecSimScript2019_01_25.m.

Reading for Next Time: Sections 2.3, 2.4, and 2.5

=================================================

Mon. 1/28/2019: There were questions about the Discussion on p. 8, which will be revised. Students worked Problems 2.3 and 2.4 in class.

HW #3: Do Problems 2.5 and 2.6 in ECE830notes2019_01_25.pdf.

Reading for Next Time: In ECE830notes2019_01_28.pdf read the introductory material about Linear Estimation in Section 2.6 on pp. 12-14 (skip Example 2.8). The material on pp. 7-8 has been revised, in case you found the original version confusing.

=================================================

Wed. 1/30/2019: No class due to weather.

Reading for Next Time: Read the rest of Section 2.6, including Subsection 2.6.1. If there is anything you don't understand, you can email me or make a note to ask me on Friday. Take a look at Problems 2.7-2.11 see if you can do some of them (not to turn in).

=================================================

Fri. 2/1/2019: Discuss Section 2.6 Linear Estimation. Showed that if $A_{1}$ and $A_{2}$ both solve $A C_{Y} = C_{X Y}$ , then the corresponding estimators are equal with probability one (see the updated sentence above the Remark on p. 13 and Problem 2.12) in ECE830notes2019_02_01.pdf). Discussed the Kalman filter & example (pp. 17-20).

HW #4: Problems 2.9, 2.10, 2.13, 2.14(you just want a single RECURSIVE equation for ${\hat{X}}_{t | t}$ and a single RECURSIVE equation for $Γ (t | t)$ ).

Reading for Next Time: Read about the Kalman filter on pp. 17-20.

=================================================

Mon. 2/4/2019: We discussed how the Wiener filter theory can accommodate any relationship between X_t and Y_t, and further, any way of generating X_t. In contrast, the Kalman filter requires a linear dynamical system for generating X_t and requires Y_t be affinely related to X_t.

HW #5: Problem 2.16.

Reading for Next Time: Read Section 2.7.2 Derivation of the Kalman Filter.

=================================================

Wed. 2/6/2019: Discussed Problem 2.16 & a typo was noted. Went over the rest of the derivation of the Kalman filter. Discussed Problem 2.15 and details of the example in Section 2.7.1. Note that a new figure (Fig. 2.5) has been added to the latest version of the class notes showing the error variance for each component of X_t: ECE830notes2019_02_06.pdf. Also the equation in Problem 2.16 has been fixed and expanded.

HW #6: Problem 2.15 in the Feb. 6 version of the notes. Use the Publish feature to create the pdf to submit your HW. Be sure to edit the publishing options to set the output to pdf rather than the default html.

=================================================

Fri. 2/8/2019: Introduced estimation of parameters.

Reading: Sections 3.1-3.4

HW #7: Problem 3.1(a)(b) in ECEnotes2019_02_09.pdf.

=================================================

Mon. 2/11/2019: Introduced exponential families and related them to maximum-likelihood estimation.

Read for Next Time: Subsection 3.4.1 Exponential Families in ECEnotes2019_02_12.pdf.

=================================================

Wed. 2/13/2019: More discussion of MLE.

Reading: In ECEnotes2019_02_13.pdf, read the new Example 3.7, which shows that solving the likelihood equation (setting the derivative of the log likelihood equal to 0) does NOT always work. Review Example 3.8, which shows that for some values of x, the MLE may not exist, even for an exponential family.

HW #8: In ECEnotes2019_02_13.pdf, 3.2(a)(b)(c), 3.4.

=================================================

Fri. 2/15/2019: Discussed estimating E[X] and trying to decide if it is positive based on simulation. Try running the script PositiveExpectationSimulationScript2019_02_15.m (nothing to turn in, but we can discuss on Monday).

HW #9: 3.3.

=================================================

Mon. 2/18/2019.

=================================================

Wed. 2/20/2019: Discussed EM Algorithm.

Reading: Section 3.4.2 in ece830notes2019_02_20.pdf.

HW #10: Problem 3.5(a)(b)(c) in ece830notes2019_02_20.pdf.

=================================================

Fri. 2/22/2019: The restriction made on Wed. that $p_{θ} (y | z)$ not depend on $θ$ can be removed. I gave an alternative derivation today, which can be found in the latest version of the notes, ece830notes2019_02_22.pdf on pp. 36-37. It is NOT necessary to read the new derivation. The key point I want to emphasize is that the boxed formulas on p. 37 are what we will be using. IN particular, you will need the boxed formula for $Q (θ, θ^{'})$ on p. 37 to do the reading below.

Reading: Example 3.14 (the mixture density example I started in class) on pp. 38-39.

HW #11: Download the Matlab script EMalgorithmSimpleMixture2019_02_22.m. Add your name to the first line. Experiment by changing some of the variables such as theta and told and running the script. The publish the script as a pdf (edit the publishing options) and upload your result.

=================================================

Mon. 2/25/2019: Pointed out errors in Example 3.14. The errors are fixed in ece830notes2019_02_25.pdf, where I also discussed Examples 3.15 and 3.16.

Reading: In ece830notes2019_02_25.pdf, read the updated Example 3.14 and the new Examples 3.15 and 3.16.

HW #12: Suppose $Y_{1}, Y_{2}, Y_{3}$ are independent with each having the mixture density of Example 3.15. Assume that $β_{1}, \dots, β_{K}$ are known and that $ψ_{k} (y | τ_{k}) = e^{- (y - τ_{k})^{2} / 2} / \sqrt{2 π}$ . Derive the EM algorithm recursion for $θ = [τ_{1}, \dots, τ_{K}]^{T}$ .

=================================================

Wed. 2/27/2019: Discussed EM algorithm HW #12. Started discussion of UMVU estimators, sufficient statistics, and the factorization theorem.

Reading: Section 3.5, pp. 43-45 in ece830notes2019_02_27.pdf.

HW #13: Problems 3.7, 3.8.

=================================================

Fri. 3/1/2019: Discussed sufficient statistics, dimensionality reduction, complete statistics, Rao-Blackwell Theorem, Lehmann-Scheffe Theorem.

Reading: In ece830notes2019_03_01.pdf, read Example 3.18 on dimensionality reduction. Review the statement and proof of the Rao-Blackwell Theorem 3.19. Read Section 3.5.2 What is a Complete Statistic?

HW #14: Problems 3.2(e), 3.9, 3.10.

=================================================

Mon. 3/4/2019: Discussed minimal sufficient statistics. Discussed the fact that "most" exponential families have complete, minimal, sufficient statistics.

HW #15: Problem 3.11.

Reading for next time: In ece830notes2019_03_04.pdf, read pp. 46-47 on minimal sufficient statistics. On p. 50, read Example 3.26 (Completeness of Uniform Densities). If you are interested in more insight about the notion of a complete statistic, read Note 3.1 on p. 5.

=================================================

Wed. 3/6/2019: Discussed complete & minimal statistics. Revisited the updated Lehmann-Scheffe Theorem. Mentioned corrections to Example 3.23.

HW #16: Problems 3.12 and 3.13 in ece830notes2019_03_06.pdf.

Reading: In ece830notes2019_03_06.pdf, read Theorem 3.26 (Completeness & Minimality) and its proof on p. 50. Read Example 3.28 (Estimation of Poisson Probabilities) on pp. 51-52.

=================================================

Fri. 3/8/2019: Discussed Theorem 3.26, the range of a function, open sets, interior points. Introduced the Cramer-Rao Lower Bound (CRLB) and the information inequality for a scalar parameter.

Reading: In ece830notes2019_03_08.pdf, read new footnotes on p. 47. Then read Section 4.1 in Chapter 4.

HW #17: Problems 3.14, 3.15, 4.1, 4.2 in ece830notes2019_03_08.pdf. The code for Problem 3.15 can be found here.

=================================================

Mon. 3/11/2019: Showed that $φ^{'} (θ)$ is equal to the expectation on the left in (4.1). Also derived an alternative formula for $J (θ)$ using second derivatives. Worked Example 4.1 Phase Estimation and Example 4.2 DC Level in White Gaussian Noise.

Reading: In ece830notes2019_03_11.pdf, read last line on p. 56-bottom of p. 57. Then read Example 4.1 on pp. 58-59, and then Section 4.1.1, including Example 4.2, on pp. 59-60.

HW #18: Problems 4.3(deleted), 4.4 in ece830notes2019_03_11.pdf.

=================================================

Wed. 3/13/2019: Discussed curvature, divergence & Fisher information, and started the vector version of the information inequality and CRLB.

Reading: In ece830notes2019_03_13.pdf, read new Example 4.1 (Curvature) on p. 58, then pp. 60-first display on p. 62.

HW #19: Problem 4.3(NEW).

=================================================

Fri. 3/15/2019: Finished deriving the vector form of the information inequality and Cramer-Rao lower bound on pp. 62-64

Review Problems: In ece830notes2019_03_18.pdf, 3.16, 4.5, 4.6, 4.7, 4.8, 4.9. Rework old HW too.

=================================================

Mon. 3/25/2019: Introduced the detection problem.

Reading (for FRIDAY): In ece830notes2019_03_25.pdf, read new footnote $a$ on p. 43. Then read Sections 5.1 and 5.2.

Review Problem: 4.9(c). See new Hints in 4.7(f).

=================================================

Wed. 3/27/2019: Answered questions. Posted new review problem 4.10 in ece830notes2019_03_26.pdf.

5:15-6:30 Exam 1. Solutions in 19se1wSolutions.pdf.

Exam grades:
97-99 (4)
89 (1)
79 (1)
50 (1)

=================================================

Fri. 3/29/2019: Discussed relationship between estimation and detection. Computed likelihood ratio for Gaussian detection problem.

Reading: In ece830notes2019_03_29.pdf, read the new Example 5.2. In Example 5.4, read the new 2nd paragraph. Read the new Example 5.6 (which is Gaussian detection problem I did in class).

HW #20: Problems 5.1, 5.2.

=================================================

Mon. 4/1/2019: Discussed analysis of $y (1 \pm y) \binom{\binom{H_{1}}{≷}}{H_{0}} η^{'}$ .

Reading: In ece830notes2019_04_01.pdf, on p. 75, there are two new sentences at the end of the first paragraph of Example 5.4. At the bottom of p. 77 and top of p. 78, there is a third remark about the scalar case of Example 5.6.

HW #21: Problems 5.3, 5.4.

=================================================

Wed. 4/3/2019: Discussed the Neyman-Pearson Lemma.

Reading: In ece830notes2019_04_03.pdf, read Section 5.3 (not really needed to do Problem 5.5 below). Section 5.3.2 works out the simpler, nonrandomized version of the Neyman-Pearson Lemma that I went through in class today. The derivation in class can be found in Section 5.3.3.

HW #22: Problem 5.5.

=================================================

Fri. 4/5/2019: No class due to EXPO.

=================================================

Mon. 4/8/2019: Compared and contrasted situations in which priors are unknown; e.g., radar detection, where the Neyman-Pearson approach is appropriate, and situations in which priors are reasonable to assume; e.g., communication systems, where the MAP rule is appropriate. Reviewed Example 5.6 with change in notation.

HW #23: Problems 5.7, 5.8 in ece830notes2019_04_07.pdf.

=================================================

Wed. 4/10/2019: Introduced the Generalized Likelihood Ratio Test (GLRT).

HW #24: Problem 5.12.

Reading: In ece830notes2019_04_10.pdf, read Section 5.4 The Generalized Likelihood Ratio Test.

=================================================

Fri. 4/12/2019: Pointed out that Problem 5.7 has a third remark and the Problem 5.8(b) is new. Discussed Example 5.9 in ece830notes2019_04_13.pdf.

HW #25: Problem 5.15.

Reading: Example 5.9.

=================================================

Mon. 4/15/2019: Discussed UMP tests, problems of density estimation.

Reading: Section 5.4.1 Uniformly Most Powerful Tests on p. 84 in ece830notes2019_04_15.pdf.

HW #26: Problem 5.16.

=================================================

Wed. 4/17/2019: Discussed UMP vs. GLRT. Introduced the problem of detection a signal of unknown delay in AWGN. Students started solving it in class.

Reading: Example 5.10 in ece830notes2019_04_17.pdf.

HW #27: Problem 5.17.

=================================================

Fri. 4/19/2019: Discussed UMP, failure detection, sequential probability ratio test, convergence of the likelihood ratio to the divergence. Importance sampling & the likelihood ratio. The GLRT and the linear model.

HW #28: Problems 5.18, 5.19 in ece830notes2019_04_19.pdf.

Reading: Look over Section 5.4.3. The GLRT and the Linear Model.

=================================================

Mon. 4/22/2019: Discussed footnotes e and f in Chapter 5. Discussed Linear Model Problem and modification for energy constraint. Discussed simulation, importance sampling, and connection to confidence intervals; estimation of something you know but is difficult to compute vs. estimation of something you cannot observe or know.

HW #29: Problem 5.20 in ece830notes2019_04_22.pdf.

Reading: Problem 5.21 (energy constraint) - how far can you get in this problem? We will discuss it on Wed.

=================================================

Wed. 4/24/2019: In ece830notes2019_04_24.pdf, discussed new UMP Example 5.8 and its implications on pp. 84-85. Discussed new Example 5.12 (Energy Constraint) on pp. 89-90.

Reading: Example 5.8 and its implications on pp. 84-85. Example 5.10 (updated with new figure). Example 5.12 (last paragraph contains additional information used in HW #30 below).

HW #30: Problem 5.21 (updated). You will need the Matlab scripts P5_21script.m, qcls.m, and makehistn.m.

=================================================

Fri 4/26/2019: Introduced Kernel Density Estimation.

HW #31: Download the scripts KdensEstScript.m and KdensEst.m, which are in the MatlabScripts folder. Run KdensEstScript.m as given using the uniform kernel. Run it again for the other three kernels provided. Then change the statement "uni = true;" to "uni = false" so that the data no longer comes from the uniform distribution. Run the script for each of the 4 kernels using the new data. Upload a pdf with all of the graphs you generate (feel free to modify the script to do all of this in a single run so that it is easy to use the Publish feature of Matlab).

Reading: Continue experimenting with the KdensEstScript.m by modifying the choice of the bandwidth h.

=================================================

Mon. 4/29/2019: Briefly discussed histograms. Discussed kernel density estimates --- mean, bias, variance.

Reading: pp. 102-top of 106 in ece830notes2019_04_30.pdf.

=================================================

Wed. 5/1/2019: Tried to motivate kernel density estimation with the empirical cumulative distribution function. Finished derivation of Silverman's rule of thumb.

Optional Reading: In ece830notes2019_05_01.pdf, you can read about the empirical cumulative distribution function. You can also read the new section 6.2.5 on the Epanechnikov kernel.

Review Problems: 3.18, 5.24, 5.25, 6.1, 6.2, 6.3, 6.4.

=================================================

Fri. 5/3/2019: Did teaching evaluations. Discussed aspects of Epanechnikov kernel. Threshold for LRT in the Bayesian case with known priors and cost structure.

Latest notes: ece830notes2019_05_03.pdf. Additional problems have been added to the above list.

=================================================